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INTRODUCTION 



The set of mflterUla. evolved as Evaluating Instructional Programa , waa 
developed under the broader funding of "Practlcum for Inatrucf.lonal 
Developers" (Project No. R020520, Grant No. OEG- 0-72 -4329). The putpoae 
of these naterlala waa to prepare indlvlduala In the competent trial and 
revision of newly developed Inatructlonal products. The ouiterlals were 
planned In a largely aelf-lnatructlonal format to foster their eventual 
dlasemlnatlon. They were designed for use by grad\iate studenta, teachers 
and curriculum apeclallsta who were to assume responsibility for formative 
evaluation of Inatructlon. Ideally, the materials would articulate with 
Research-Baaed Techniques for Instructional Design (Baker and Quellmal2» 
I972y, developed under the auspices of the national Center for Educational 
Research and Development, U.S. Offlce«of Education. 

What follows will be an unelaborated treatment of the development history 
of Evaluating Instructional Programs . Including the decisions made and 
options foregone aa the product came Into being. Vlhile there Is always 
faint hope that a process In retrospect will serve as a positive example 
for future developers, e*ils report will attempt to Include the algnlf leant 
aspects of tlte-sdevelopmeat activity, both good and bad. 

The need for a Mit of materials to teach formative evaluation techniques 
%ias apparent at cfae time of proposal and continues to this time largely 
unsatisfied by efftwrta elsewhere. While there have been rules and guide- 
lines produced tha^ suggest data gathering and analyals techniques, they 
tend to be Inadeqinte for a number of serious reasons: 

1. They ^mpp^Bt to savor dealing technically with problems 
thatrmlght^'^r^ated more almply. 

2. They <io notVlghllght data gathering for the purpose of 
Improving instruction but rather focus on the full descrlp- 
tl^ of wha^ a program does. 

3. ^ey often do not fall within an established frame of 
reference for development activity. 



4. None have been subjected to empirical tryout and revision 
processes themselves. 

The designer of th^ materials also had a long hlatory of teaching courses 
In Instructional development and, from that experience, could anticipate 
the claaa of problems of concern to novices In the area. Thus, the mate- 
rials In the early planning stage were designed to meet the following sets 
of needs: 

1. To present clearly and simply Information that would help 
Individuals plan formative evaluation trials. 





2. To emphasize that data gathering should be Instrumental to 

the Improvement of Instruction and not a rite In and of Itself. 

3. To build In opportunities for l^idlvlduals to work with slmu- 
l^ated data. 

* *• . ■ 

4. To Integrate Information about revision of programs with the 
data, collection/analysis activity. 

, 5. TO present problems that had a strong flavor of verisimilitude 
/ In an attempt to reduce the abstract, and unreal character that 
"practice" exercises often share. 

6. To design materials that subjects will not mind using, by In- 
corporating the reactions of subjects to the materials 'In the 
revision process. 

The objectives of the program were directly formulated and remarkably 
stable throughout the development process. They were as follows: 

After completing the materials, the learner would be able to: 

- Identify relevant data sou -es for formative evaluation 
and to outline a plan th-' ncorporated appropriate data 
sources. 

- Display data from a variety of sources, to sunnarlze Informa- 
tion In non-statlatlcal ways and to draw Inferences about 
program components that require revision. 

- Suggest revisions of Instructional materials when provided 
with a program description and an unanalyzed data set. • 

The major concepts treated In the program were as follows: 

1. Collect the least amount of data required for Improving 
Instruction limiting the sample size and the Investment 
In slick materials at early stages. 

2. Attend to more than prespeclfled cognitive outcaaes. 

3. Display data visually rather than only suinnarlze atatlstlcally 
to foster understanding of what Is happening in the program. 

4. Revise programs consistent with the extant data base In re- 
search on Instruction: focusing first on the use of Instruc- 
tional principles, second, on alternative methods, and last, 
on alternative biedla. 

This reconmended sequence also Is scheduled In terms of increasing cost 
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DBVELOPMQIT FUN 

Although this project was planned to bo pursued Independently >f the major 
•TPractlcwn" effort, the Interaction of the two projects produced both 
positive snd negative consequences. One seriotxs difficulty that emerged. 
In retrospect, was the shared funding between the two projects. Project 
clerical assistance, for example, wrs often detained by work on the larger 
activity (which ultlssat - had stricter deadlines becauae full claasroom 
groups were employed long periods of time). The staff of Evaluating 

Instructional Prograae m very thin In numbers, never Involving more 
than two half- time aaslatants, for one three month period^ and usually 
consisting of one half-time research Assistant, plus the principal in- 
vestigator. Even so. those Individuals were sometimes Interrupted in 
their activity in order to assist the Practicum staff in their develop- 
ment effort. The consequences of these interruptions resulted in sn 
extraordinarily uneven development activity, even more so than normally 
induced by the "hurry-up and wait" problem endemic to the generation of 
Instructional material. 



Schedule 

The achedule for development of the project proceed -hI in the following 
broad categories: 1) Content development, 2) Product design, 3) Objectives 
and criterion measure refinement, 4) Exerciae development, 5) Text pro- 
duction, and 6) Component integration. 

Figjre 1 
Schedule of Development Events 



Fall, 1972-73 
Winter, 1973 



Spring, 1973 
Sumer, 1973 



Fall, 1973 



\ 



Development of objectives. 

Tryout oi. content through series of oral 
presentations (UCLA course and AERA). 

Preliminary exercise and measure development 
tryout (AERA). 

Revision of exercises and meaaures. , 

Development of text, exercises, reflnonent of 
meaaures. 

Addition of section on baalc atatlstlca. 

First trial of partially integrated package (Florida). 

Revision of materials, preparation of additional 
text material. 

Dlatributlon for tryout at remote Practicum aitea 
(Indiana, Oregon, Arizona). 



winter, 1974 / Tryout of materials on curriculum specialists 

(Lawndale). Revision. 

.' Tryout on curriculum specialists (Pomona). 
Revision. 

Spring, 1974 Tryout on reading specialist candidates (UCLA). 

Dissemination, at AERA training session on 
Formative Evaluation, 

Corrections 6 

Preparation of Pinal Report. 



Development of Dependent Measures 

The dependent measure problem for development activity is particularly 
vexing. On the one hani), we have, been taught, and teach others, that 
adequate samples of criterion performance are Important. On the other 
han^d, ve have concerns for the well-being and good humor of those acting 
a F subjects for us. We do not wish to present test batteries that are 
longer than the instruction Itself. Nor does the solution of item sampling 
meet our needs. While distributing alternative sets of ita.^8 to diffar- 
ent learners is an acceptable solution to test fatigue and idiosyncratic 
item problems, in the process of formative evaluatii.on, its applicqbility 
is limited by the small numbers of individuals who participate in any one 
trial. Our sample was further limited by the specialized nature of the 
content. We did not think it fair to use the old standby sorts of sub- 
jects (such as students enrolled in Psychology 1) as our population. 

Our decision was to use tests that were embedded in the instructional 
materials as an indication of learner success. Utese items, designated 
"tests,*' v/ere admini8::ered in the context of Instruction and seemed to 
be reasonable extensions of the work in \A\ic\\ students were engaged. 
Certainly the threat to validity of sequence prompts can be raised, but 
our overall concern was reducing the salience >^f "testing" in the tryout 
of the materials. The dependent measures, then consisted of criterion 
situations that required the students to Integrate information treated 
in each component. . 

Development of Content 

Our experience with the planned content of the program led us to assume 
that previous treatments of data collection and revision of instructional 
materials had to be presented more simply (less technically), and in a 
manner which linked at the outset the kinds of data collected to the 
range of decisions and improvements possible in a set of instructional 
materials. In order to obtain feedback on the "principles" or guidelines 
that were to be taught by the materials, oral presentations were made by 
the principal author. The first of these presentations was an extended 



lecture given to etudents In the Learning and inatruction program at UCU 
during the Winter quarter of 1973. Baaed upon the reactions and cdamentfi 
of th»« students, the presentation was revised, principally to add soae 
dlscutislon of pretesting. ' The presentation was again presented at a Mini 
Training Course offered at the annual meeting of the American Educational 
Research Association In New Orleans. The topic of the training session 
was "Formative Evaluation of Inatructlon" and the session was very well 
attended (N»70f). Further refinement of the, conceptual base of the prtiduct 
was made «s s consequence of these trials. These presentations alao per- 
mitted the Investigator to ascertain seme of the dealrable prerequisites 
of the target population. Our feedback from these sessions was generally 
positive. We were repeatedly told that concepta treated were valuable 
and unavailable In other Instructional formats. The riiports of the AERA 
Standing Comlttee on Training, as derived from trainee evaluation of 
the aesalon, were poaltlve. The Coenlttee approved a second training 
session on the same topic l>y essentially the same staff which waa offered 
m 1974. 

Product Design Activities \^ 

The materials were constructed with attention to research-based Instruc- ' 
tlonal techniques (Baker and Quellmalz, 1972; Holland, Doran, and Frezza, 
1974) Ample opportunity to practice criterion behavior was included. 
Clear 'descriptions of tas^ji and feedback were provided for the learner. 
The. sequence of activities both within and between aectlons was charac- 
terised* by a reduction of pronpts. . \ 

Beyond the studied use of' such research-derived techniques, the flow of 
materials was to present Information, to present opportunities for 
discrimination, and to provide opportunltlea to practice. Concurrently, 
a reverse strategy was being tried out In the major Practlcum activity, 
where students were presented with simulations at the outsat, designed 
to stimulate Interest and to provide an organizing ^structure for subse- 
quent activity. Although the taaks treated by the ••Practicus" project 
«nd Bvaluatinn Inatructlonal Programs were distinct, and thefte 
considerable instructor effect in the Practlcum program, the general 
aequence of ualng simulations at the outset did not prove successful, 
and revisions were made that incorporated a deductive strategy almilar 
to that employed in the materiala described in this report. 

Practice exercises were produced for the materials by Individuals who 
had strong backgrounds in research in paychology, teaching experience 
at either elementary and secondary levela, and experience in developing 
materials for use by teachers. The latter two attributes made these 
Individuals particularly attentive to developing exerclaes that were 
mtereatlng in and of themaelves. In addition, a content-age l«vel 
matrix waa generated to assure that topics used in exercises sampled 
across the following areas: 



0^ . 



1 



1. Age of learner .« 

2. Cleatroon or non- classroom use of product 

3. Large or small scale development 

A. Science, social science, humanities as content areas 

5. Affective aa well aa cognitive goals 

Although there were many prerequisite content areaa upon which the product 
waa based, there was an attempt to update learners* knowledge in these 
areaa incidentally rather than directly. For example, the product contains 
no extended treatment regarding the areas of instructional objectivea, task 
analysis, or domain referenced achievement teating models. However, In the 
exercises employed, objectives presented are operationally stated, atate- 
ments of sub- tasks are included In the revision exercises, and examplea of 
domain-statements are provided in the revision exerciaea aa well. 



Product Format 

The design of the product waa predicated on a print format, primarily be- 
cause of cost snd the aasumption that present modes of dissenination favor 
print. During an early tryout, a series of cards was used to permit indl- 
vlduala to sequence exercise items of interest, but such a format proved 
cumberaome and was discarded in favor of a notebook. While the notebook 
format waa feaaible for early development and tryout activity (the note* 
books could be refilled), cost wculd prohibit wider dissemination in that 
mode. In addition, as a caaual affective gesture, photographa were inr- 
aerted in the text for the laat two empirical trlala. The positive re- 
aponae generated by theae photographa was dramatic and they are maintained 
in the final copy. Hateriala have been color coded by aection to facili- 
tate use. 



Adaptations of Criterion 

During the development effort, modifications were made in nunber and format 
of criterion itema, alwaya in the service of providing better coordination 
between what we aaked atudents to do and what we hoped the program would 
achieve, Thia fact of aucceasive item reviaion, a rather conatant feature 
of development activity of any aort, reduces the clarity with frfiidi per- 
formance increments from tryout to tryout ^csn be Inferred. The goala of 
the t>rogram remained constant, yet aufflcient modifications in the criterion 
meaaurea occurred to make comparability among triala Impressionistic rather 
than deciaive. 

In addition, during the f.ryout we collected information regarding student 
reaction to the Ideaa conveyed by .the mater iala, the rule and format of 
the materials, the stud#mts' perception of the usefulness of the materiala. 
In addition, we aolicitad, through queationnaire and Interview, students' . 
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vlewL of the materials In an attempt to determine if unintended percejp 
tlon*- were encouraged by the materials. 
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COMPONENT DBSCRXPTIONS 



Cowponent Deacrlpttoni : 

The oaterials were organised In a way designed to facilitate acquisition 
'Of the objectives and subobjeetlves of the project. Below each ouijor 
aectlon Is described, keyed to the objective and subobjeetlves to which ^ 
It was directed. 
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EHPIRICAL TRTOUTS OF PRODUCTS 



The empirical tryout procedures, used In this project were Inforised by a 
number o£ considerations, Vifst, the staff wanted to have usable data 
on a schedule that would permit subsequent Inferences to be Incorporated 
In product revision. Secondly, we wished to gather Information on sub- 
jects that represented the range of tsrget population to which we were 
directing the materials. We were partially successful In both needs. 

The data gathering ventures are listed In Figure 3, 
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BEST COPY AVAILABLE 

Kacli tryout Is hrtcf lr tlcscriHoU nnd perl IiumU d.-xtn j>r»«st>nro<^ . Ono t.i 
notice how t»io Intervnl between trlflls ciHMparcd mntorinlr. revise* 

accoleratod as the projoct drew to a close. 



Tryout I. UCLA 

The conceptual tryout of the materials took place within a course context. 
Students had Seen Introduced to the basic Practlcun materials In the Fall 
quarter, but the course had ended before the topic of formative evaluation 
had been tested. The instructor (the present author) presented the notion 
of formative evaluation, data sources for prototype and field tests, the 
ideas of data display, and principles for revision In a two class session 
(apnroxinately four hours). The presentation was Interrupted by questions 
anc^' comnents bv students. On the basis of such remarks, the Instructor 
made some modifications In organization of concepts and added the Idea 
of pretesting (which had been an unstated assumption). In addition, 
students tried out the initial set of fifteen practice exercises developed 
for Section I (data .sources) for the project. Their role was to criticize 
material, detect ambiguities, and suggest corrections in exercises. 



Tryout ?.. American Educational Research Association 

In a Mini-training session conducted under the auspltes of the Committee 
on Research Training,, the author and staff presented the concepts underlying 
the program to an enormous crowd, jammed in a small room. Each participant 
paid eight dollars to AERA for the privilege-'of attending the session. 
>iore than eighty individuals began the four hour period, but some left 
because of the scarcity of materials (only 50. were prepared) and crowded 
conflitlons. Drs. Evan Keislar and llerl Wittroch, of the UCU prngran on 
Lcarnlnr and Instruction and the Practlcun staff pnd Dr. Edys Quellpalz, 
of the Southwest Regional Laboratory p,-rtlclpated with the author in the 
tryout. After an Introduction to evaluation, Drs. Baker and Quellmalz, 
presented the necessary Information on data sources for formative evalua- 
tion of Instruction. The participants then worked in small groups on certain 
of -he e-erclses for Section I. Staff members observed and collected 
i^r^yp^^tion regarding points of difficulty. Tlic session met with apparent 
success as a second offering was scheduled for the next annual meeting 
(1974) of AERA No formal data of an achievement sort was collected. A 
brief questionnaire was circulated to the group but reponses were Irregular 
and focused mostly on the fact that the room situation was poor. 

Tryout 3. Florida A & M University, Tallahassee 

Durln'5 the Spring, the project was contacted by Dr. Wally Cox of Florida 
A 6 University, regarding materials for a workshop he was conducting on 
evaluation. Our staff agreed to supply him with about one day's worth r.f 
material in the area of instructional evaluation. His group was antici- 
pated to be around 30 individuals, much larger than what we would have 
selected for a first formal tryout. However, we agreed to the project. 
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Dr. Arlene Fink, of the UCLA Center for the Study of fivaluarioa, ha.l planned 
to partlcipatt? In the v^orkshop using Center materials. She af>;roec1 to modify 
her schedule and to monitor our part of the tryout far us, thus raving * " 
I V.I vol funds. Dr. Fink prepared a report suinmar l7.lng her activities, her 
observations of the class and suggested possible ways to interpret the 
data. . . 

The 2^ participants Included 16 public school teachers and 13 administra- 
tors. AH came from the Florida region and were distributed in elementary' 
and secondary school posts. The procedures of the tryout were as follows: 

Introduction, 15 minutes •: 

Pretest, Part I, 15 minutes 

Data sources, 1 hour 

Posttest, Part I, 15 minutes 

Pretest: Appendix (Data Analysis), 15 minutes 
Appendix, 1 1/2 hours 
Posttest, 20 minutes 
LUNCH 

Pretest, Part II, Revising Programs, 15 minutes 

Part III, 1 hour, 50 minutes 

Posttest, Revising Programs, 25 minutes 

No more than three-minute schedule overruns were permitted and no addi- 
tional breaks were given. Although materials were Intended to be self- 
instructional, groups of subjects worked together to clarify written 
materials and to share problems and strategies In completing the practice 
exercises. Such cooperation was fostered because the subjects were crowded 
and the previous three days of the Institute had accustomed participants 
to working together. In addition to completing the pre and posttests the 
participants filled In a self-description sheet regarding background 
Information and a post Instruction attitude Instrument, They were also 
encouraged to write notes and make comments In their notebooks. 

.Materials were presented in a notebook format, with each section color 
ceded. Because of time pressure, material was reproduced by spirit 
master and was loathsome purple-print. 

Results 

Subtests for each section contained six Items. In Section I students were 
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nsked to Identify nppropriate data sources, in Section II tc^ solve simple 
co.,nputational problems in data summary, and In Section HI to determine 
vhethoV revlsCfcn was necessary or not when presented with product des- 
criptions and data to analyze. • 

Table 1 

^\ Pre. pnd Post Test Scores: Florida A & M 

PRETEST POSTTEST 
SectioiT^I^— - 187o ^3% 
Section II 307, 367. 
Section III 377. 337. 

The strange, and disappointing results, particularly for Section HI had 
.some explanation. Student participation and completion of exercise 
material was reasonably good for Section I, 21 of 29 students completinj; 
the materials. For Section II, 13 of 29 completed the materials, and for 
Section III, only 8 people completed the last exercises. Subjects also 
commented profusely on the post,test items for Section HI, with indica- 
tions that they did not understand what was expected of then. Dr. t ink 
also noted that the compressed treatment time was a problem for many of 
the «?tudents who expressed the need to think over some of the material 
presented rather than charge tnrough the notebooks during one day. 

Attitude information collected was also puzzling. Given poor posttest 
performance, the attitude data was unusually good. All 29 individuals 
indicated that they felt the materials would be useful in their job 
situation. Of the 23 subjects completing the attitude Instrument, 21 felt 
section I was helpful, 19 thought Section II (Appendix) was helpful, and 
• 20 thought section III was helpful.- Negative comments centered about the 
large number of practice exercises and confusing directions and language 
in the text. Most frequent comments of « j:onstructive nature focused on 
the need for more time, and the suggestion that group discussion be inter- 
polated with reading and exercise material. 



Revision 



Following Trial 3, the following revisions were instituted: 

1 Content related to field testing was dropped; emphasis was now 
on instructional prototype tests, as examples of formative evalua 
tlon with the most serious and interesting problems for the 
potential audience of the book. 

2. Clarification of directions for exercises and tests. 
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3, Clarified and reduced iiumhcrs of practice oxt'tcisos for 
Soct l.ms I and [II. 

Movoli Dnta Stimnnry section to Appon<Hx an*! <li»lot(Ml pro rtml 
post tests. 

5. Prepared test for Section !!• 

ft. Edited and simplified language throughout. 

7« Revised all pre and postteats. 

3. Improved attitude survey so responses would be more specific. 
Trial 4. Los Angeles 

In anticipation of planned tryouts at the four sites where the Practlcum 
project was being tried out, some of the revisions were tested on a small 
sample (N«2) of teachers in the Los Angelas area. These teachers went 
through the materials, took all tests and commented profusely on the 
notebooks. Inferences from the data were difficult to make in that one 
individual did extremely well throughout, while the other had continued 
difficulty. Neither of the Individuals appeared to like the materials, 
and comni^uts relatied to boredom were scrawled on the materials. Parti- 
cipants seemed to feel there werb too many examples Included in the text 
materials and that too many practice exercises had been produced^ Each 
of the teachers had read materials in one unreli-eved sitting and thus 
feelings of boredom might have been exacerbated by the compressed treat- 
ment and lack of socialization. Nonetheless, the staff was considerably 
depressed at this point, and introduced into the subsequent versions » 
the directions for subjects to bypass exercises when they felt they 
understood the concepts. 



Trial 3« Oregon, Indiana, Arizona 

These trials were anticipated for students In the Practlcum couraes being 
simultaneously offered at the University of Oregon, Arizona State Univer- 
sity and Indiana University. Revision of materials and necessary repro- 
duction In pteparatlon for such'^^youts occupied a great deal of time. 
Materials were mailed as schedulea well In advance. Unfortunately, due 
to the exigencies of course organization, tKe materials were apparently 
not seriously, required at any site. Moa^ devastating for us, however, 
was the fact that they were not returned jit^omptly. We expected these 
trials to provide us with high quality Information. Instead, we 
received only partial data from Individual students. We also received 
a number of Individual letters from students Indicating that they felt 
positively about the materials and would like to have copies of the 
materials made available to them at a later time. However, the tragedy 
for the project was that after considerable expenditures, we did not 
have reasonable information abiut whether the materials, as revised. 
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worked any better. 

Motebooke were dlatrlbuted to the Prectlcun altes to be edmlntatered by 
the professor In hie regular course. Data were returned fro« only aeven 
of the 35 notebooks prepared. Reaults for four of six Individuals were 
received from the Indlsna trial, but only three individual a returned 
materiala from Arizona and no uaable information came from Oregon. 



COKPONEMT 



Data 

Sourcea 

Revising 
Programs 



INDIANA 



N 
4 



PRETEST 
(18)* 

6.25 

(22) 

0.00 



POSTTBST 
(18) 

16.00 

(24) 

20.66 



M 
3 



ARIZONA 

PRETEST POSTTEST 

(18) (18) 

4.33 / 15.00 



no 
data 



no 
data 



* number possible. 



A follow-up questionnaire yielded only three responses. Although the re- 
sults sre especislly scanty, there ia evidence that the programa were 
controlling responses from pre to posttest. However, dependency on such 
sparae data waa auiqidal and we decided to test again. 

Trial 6. Lawndale, California 

In desperstion, a local teacher institute was set up end taught by a staff 
member at a nearby achool district. The focus of the institute wss on 
evaluation and the materials provided a baais for the courae offering. 
In January, the materiala were used by eleven secondary achool teachers. 
Their reaulta, by sections are preaented below: 



Table 2 

Performance of Lawndale Teachers 



Section I 
Section II 



PRETEST 

24Z 
NO TEST 



POSTTEST 
MO TEST 



Section III 
Select 
Impleoient 



5X 

n 



63% , 
24X 



Attitude information waa olao collected for thia trial. In interviewa, 
all participanta indicated that they could use again the concepta they 
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learned, etpecUlly the'reyltlon principles. The revision section was 
considered the nost useful pert of the material and only one of the 
eleven did not feel conpetent to plan the evaluation of Instruction. 
Program strengths were reported by participants In terms of the vslue 
of practice and feedbsck exercises, the concepts taught and appendix. 
Weakneases Identified related to writing atyle end a lack of sufficient 
examplea (the oppoalte crltlclan frcn Trial 4). The obaervera noted 
that boredom was expressed during Section I. A desire for group prsctlce 
on exercises was observed during this trlsl. The most serious problem 
^s In maintaining teacher* a attention at the time of day the workshop 
was offered (3 to 5 p.m.) after a full teaching day. 



Revision 

Some difficulty emerged In the revision section again. The material was 
seen to be of vslue to participants and they were able to Identify princi- 
ples for sppllcstlon bssed on the analyala of data. However, a difficulty 
. related to the extent to «4tlch they could provide adequate deaerlptlons 
of how such principles would be Implestented. One might ratloriailse this 
difficulty bssed upon the fact that they were given very abbreviated 
descriptions of hypothetical materials and thua their Implementation 
might tend to be general rather than apeclflc. The test Itself wss 
scrutinised, and a thorough analyala of pre to poatteat Itema on the 
revlalon aectlon dcmonatrated that there vaa unequal difficulty between i 
the two tests. The pretest provided a altuation with two objectlvea and / 
two subobjectlves for analyala, while the poatteat asked for th^ee objec-/ 
tlves snd three subobjectlves. On the pretest examplea, all data were / 
hottogeneous scross data aourcea, while on the poatteat a number of con- / 
fllcts were Introduced. On the pretest, ten of the elev<sn subjects were/ 
unsble to suggest even one correct revision sctlvlty, whereas, after / 
Instruction, all were sbla to suggest at leaat one, and moat three or 
moi^ appropriate prlnclplea of revlalon. 

Following the data analysis, the program waa aubstantlally revised. 

1. Section XX waa expanded. 

2. Pretests snd posttesta were made parallel, 

3. Directions were clarified, 

4. Anawer sheet formats ware changed. 

5. Text waa heavily edited and simplified. 

6. Text aectlon was visually refonssted. 

7. Photographa with captlofia (for affect) were added. 

8. Glossary waa added, " 

9. Teacher guidelines were prepared. 
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BEST COPY AtfSlLABLE 

Trial 7. Pomona 

:iovrntiMMt curriculum .spccinllstK provMoil »1atn rr(M,i a trial conOuvte^l hy 
a>,rai1uate of the lia>A doctoral proj'ratn. iV/enl^^-two subiects partivipnLed 
in the tryout, eleven reading and eleven ma thoim t Ics specialists. Only 
17 participants completed the workshop. The workshop was scheduled on Iwo 
half day periods. Report of the results is presented below: 

COMPONE^NT - N • PRETEST N PliETEST 

Data ^ 
Sources 17 . 2.93 17 13.35 

(2S) (3?.) 

Revising 

Pra^rans 17 17.32 14-16 31.33 

Affective data were obtained through the use of a que-gtio^naire. Parti- 
cipants were apparently satisfied with the materials, a departure from 
pervious tryouts. Of the 15 responses to this questionnaire, the average 
rating. was 2.82 on a 1-4 scale for the attention holding properties of 
the n«aterials. Positive responses (2.91) were also obtained for the 
'Appendix section. Participants rated the fairness of the tests and 
clarity of feedback especially high (3.13). ^ The average rating across 
all items in the questionnaire was 2.83. A copy of the questionnaire 
is included in the Appendix. 



Trial 8. UCLA • 



On 



Was this tryout the criterion test? We had antici(>ated that it was. 
the other hand, we felt that with minor revisions, the product wight be ^ 
substantially improved. To that end. we conducted a tryout in April 
that would simulate an alternative form of product use. Students were 
pretested nn the materials in two class sessions, allowing about 20 
minutes per session for responses. Notebooks were distributed to students 
with directions to complete all the activities and to be prepared to take- 
a posttest on the material. We attempted to make it clear that no grading 
corftlngencv was related to performance, on 'the test. At the next class 
session, after a one week interval, students completed the posttest and 
questionnaire. 

Students in the trvout were teachers studying to obtain a reading 
specialists credential and a master's degree. The results from the 
tryout are presented below In Tables 3, 4, and 5. 



Table 3 

Performance of UCLA Students In Final Tr you t 



PART I> EATA SOURCES PRETEST J POSTTEST Y ^ 

Data Sources (A)* * .98 

Subject Characteristics (2) - .28 1.56 

Instruments (5) , .67 4.44 

Administrative Problems (7) ^'^^ 

PART II. REVISION 

■ * 

; Graphing (3) ,06 ^ (A) 4.00 

Identifying principles 

of Revision (10) 1.24 7.35 

Implementing Principles (1) .00 .66 



* number possible. 

* 

students were also asked to keep records of their reading times and were 
encouraged not to attempt to assimilate the entire set of materials at 
one sitting. Results of reported reading times are presented In Table 4. 

Table 4 

Reading Times - UCLA Pinal Try out 

TEXT EXERCISES 
X 

Data Sources 47.95 minutes 50,62 minutes 

Revl6ii>n 55.43 minutes 80.00 mlnutee 

t 

Estimates obtained for section averaged about 1 hour. Therefore, the 
entire time required to complete the materials at the demonstrated level, 
of performance was under five hours ♦ 



Table 5 
Results of Attitude Survey 



Overall rating- of product (4) . - 
Concepts clearly presented 
Posttests fair • 

Directions for practice clear, easy to follow - 
Information will be useful later 3.24 

From an inspection of the data, one mlp,ht infer that the staff 
as well as relieved by, the results. 
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PEBSOMMEL AND PROJECT COSTS 



Project personnel have been limited to the principal investigator and a 
snail group of graduate atudents or reaearch assistants. Early conponents 
of exercises were prepared by Judy Safford and Saaellyn Wood during the 
Winter of 1973. Deborah Feinberg was reaponaible for the review aeetion 
on ststisticsl procedures. Both she and Lynn Smith prepared practice 
exercises and early versions of the criterion tests. The majority of 
exercise writing, revision snd dsts analyaia was executed by Aleta 
Saloutoa. Shireen Powell assisted in dsts analysis. Tha burden of 
coordination of nateriala and field teats was Hs. Saloutoa* reaponsi* 
billty, slthough MS. Smith assisted her in her tssk. Uhfortunately, the 
staff aaslgnments were thinly dispersed oyer the period of funding, 
Lynn Smith hss been with the project over the period of June through May, 
although only three months of half-time employmodt waa compeniated. 
Aleta Saloutoa Joined tha project half-time in September and continued 
until April 30 in that role. The discontinuity of aaaignmenta and moderate 
proportion of compenaated time represented minor problema. Other rolea 
tended to interfere with the project's activity, eapedally reaponaibili- 
ties associated with the Practicum and other Job cooBltmenta, Nonetheleaa 
the project held together, apparently through loyalty of the Junior ataff. 

Staff coata were small. From expendlturea reported aa of June 1, we 
eatlflute that leaa than $13,000 were apent on the deaign, development and 
testing of these materials. When we include supplies, reproduction costs, 
overhesd, and travel for dioaemlnation pur<H>aea, the project waa completed 
for conaiderably under $20,000. A groas vv^inaformation to costs by hour 
of "validated" materials would be under $2,000 for each hour of inatruction. 
Aa a coafiarison, the project by Baker and Quellmals (1972) required approxi- 
mately $22,000 in peraonnel coata for a package of 12 houra duration. 
Ap:>arently, in the face of inflation, development is becoming ali||itly 
more efMeient. 

The project waa benefited by the low overhead rate from OCLA Extenaioo, 
as well ss the "free" work voluntarily performed by the staff. The 
relatively low cost figure wmy encourage the funding of diacrete (rather 
than large acale, programmatic) development efforta at univeraity aitea. 
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DISSEMnUiTIOM 



B«rv.«lve concern of developer, li the problem of dietrlbutlon of 
S^teHarie^i^^S for .pecUlUt. rether then widely dUtrlbuted group. 
"'cS i te2:;i« it le'not elwey. eeey. even -'^iJ^J.^:^^^^ 
subjected to rigoroue eveluetlon to pureue the usually ^••^'•J^*/"^" 
c«eJn.l publication. Ideally. co«ercUl publication %<ould be the 
Ideal destination for these naterlala. 

Aa far as more Informal torts of advertlalng. aooe Intereat In '^f »«te- 
rUU hat already been created and demonatrated by letter, and other 

~ rr.^t:: r-^^^-r ^^^^^^^^^^ 

•kill*. 

^t.ri.l. have also been offered »• potontul t»»lnlng v«hlcl«» for 
The ""'^l* ^reree of dreet eduction. Ihelr greatest 

with t.e advent of -^i^^eiraUjr:^ t^'a rirt^lStl't. recog- 

ihf^UlJlS eiS^Hd in training .ere plam>.d for 

such a conclusion. 

Sj be «5"".llable btr>n the li«lted n».ber. of cople. «e are able 

to print. 



Report of Panel 8, Conference on Teaching. Hatlonsl Institute for 
Education, Washington, D.C. , June, 1974. 
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SlMMkRT 



This project produced an Integratod set of matertala chat are appropriate 
for uae in graduate training, reaearch and development settlnga, and 
public achool peraonnel uae. The naterlala were developed In the courae 
of repeated tryout and revlalon cyclea and now meet the expectatlona of . 
their dealgnera. The developattit hlatory of the materlala vaa limited 
becauae of format expectatlona and concerna for dlaaemlnatlon. Rowaiver, 
the preaant form of the materlala la apparently appropriate. 

No cataatrophea marked the development proceaa of Evaliiatlnft Inatructlonal 
Programa . The relatlonahlp of thla materlala devel(^»M»nt effort to the 
larger Praetlcum model training project Introduced aome unavoidable dlffl- 
cultlea, particularly In ataff utilisation. However, poaltlve aide ef facta 
were alao atlmulated by thla project. The activity aupported In part five 
different graduate atudcntai, one of whom waa Influenced to puraue doctoral 
work aa a function of her reaponalbllltlea on the project. The opportunity 
to work on theae materlala alao auggeated aome alternative waya of con- 
ceptuallalng reaearch on Inatructlonal rolea, methoda vAlch will be explored 
In planned reaearch by thla author. 
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INSTRUCTOR GUIDELINES 
Evaluating Instructional Progran s 



- Directions to Administer Program 

- observation Forms 

- Pre and Posttests (with answers) 

- Glossary 

\ 

] 

i 
1 
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I. DIRECTIONS FOR ADMINISTERING PROGRAM 
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GEI5ERAL DIRECTIONS FOR THE INSTRUCTOR 
(Used In Tryouts) 



p»ni... Mn.> rn.tructlonal Programs is designed t.> Sc ptlmarllv sol^- 
instr'ctional but can also be us. d cffectlval>. in a workshop or discuss.oa 
setting. The program consists of three major sections and an Appendix.^ 

Section I Data Sources for Prototype Testing 

Section II Data Summary for Revision 

section III Revising Instructional Programs 

Appendix Elementary Data Summary Procedures 

Tlie materials In each of these sections, with the exception of Data 
Summary for Revision, are accompanied by pre and posttests. Regardless 
of .the method (self- instructional or teachdr-nedlated) In which you 
present the oaterlals, you will want to 

1) remind participants to complete the Personal Inventory 
preceding the Introduction; collect the Inventory forms, 

2) remind participants to write their names on the pretest at 
the beginning of, each section; collect the pretests after 
they have been completed, 

3) remind participants to write their, names on the posttests 
at the end of each section; collect the posttest, 

4) instruct participants to reply anonymously to the ques- 
tionnaire provided at the end of the text; collect the 
questionnaires* 

After the participants have completed Section II, they should be asked 
to review the Table of Contents In the Appendix. The Appendix provides 
supplementary Instruction In the summarization of data, construction and 
Interpretation of histograms and frequency polygons. One must be able 
to apply these techniques In order to do the exercises In Section III. 
If there are topics In the Table of Contents with which an Individual 
Is unfamiliar, he or she should then read the appropriate part of the 
Appendix and complete the practice problems that accompany the explana- 
tory material. If participants are familiar with all of the topics in 
the Table of Contents, there is no need to attend to the Appendix. 

In order for this program to be of maximum effectiveness to participants, 
tT.odlflcatlons should be considered in two areas of the Instructional 
sequence: 



\ 
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1. terminology ; Unfamiliar language used in cfovelopment and 
evaluation should be explained to the participants. This 
might be done on an ''as needed** basis by the instructor, or 
In written, form by duplicating the accompanying Glossary. 

2. relevance : Special attention should be given to pointing out 
the. utility of the program to specific participant groups. 
Teachers, for instance, will benefit from learning a) to 
select materials that offer evidence of preferred development 
procedures, b) to measure empirically the effectiveness of 
Instructional materials with their own students » and c) to 
utilize research-based principles for Improvement of mate- 
rials. It is highly desirable that the Instructor take 
i^dvantage of any opportunities to emphasize the utility 

of the materials for the particular group of participants 
with whom he/she is working. 

Wo would greatly appreciate your completion of the Instructor's Observation 
Checklist, and any additional remarks which would help us Increase the 
effectiveness of the program. Your cooperation in pa-rjtlclpating In the 
use of these, materials Is sincerely appreciated by the development staff,' 
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IKSTRUCTOP.'S OIiSERVATtO\^ CHECKLIST 
Evaluntinr. Instruct tonnl Pro-.rnns 
Section I 



A. Administration of llaiertals 

1, Time taken to complete pretest 
Remarks: 



Start Finish 



Elapsed 
Tin^e 



2. Time taken to read expository 
tnaterlal 

Remarks: (ability to proceed i^lthout 
instructor aid, confusion re: terns, 
concepts) 



-3. Tine taken to conplete practice 
exercises 

Renarks: (a-)ility to complete without 
aid, frequency of referral to text) 



Time taken to cor.plete posttest 

Remarks: (frustration level, clarity 
of directions) 



3, Attitude Toward Materials 

1. Participant comments about difficulty level of material: 



2, Participant comments about utility of material: 
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INSTRUCTOR'S OBSERVATION CHECKLIST 
Eyqhiattn^ Instructional Programs 
Section II 



St.nrt Finish Elapsed 
A* Adninlstration o£ Materials Time 



1* Time taken to complete pretest 
Rei:iarks; 



2» Time taken to read expository 
material 

Remarks: (ability to proceed without 
instructor aid, confusion re: terms, 
concepts) ' 



3. Time taken to complete practice 
exercises 

- Remarks: (ability to complete without 
aid, frequency of referral to text) 



4, Time taken to complete posttest 

Remarks: (frustration level, clarity 
of directions) 



B« Attitude Toward Materials 

1* Participant comments about difficulty level of material 



2. Participant comments about utility of material: 



ERLC 
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INSTRUCTOR'S OBSERVATION CHECKLIST 

Evaluating Instructional Progrnms 
Section III 



A. Administration of Materials 

1. Time taken to complete pretest 
Remarks: 



Start Finish Elapsed 

Time 



2. Time taken to read expository 
material 

Remarks: (ability to proceed without 
instructor aid, confusion re: terms, 
concepts) 



3. Time taken to complete practice 
exercises 

Remarks: (ability to complete without 
aid, frequency of referral to text) 



4, Time taken to complete post test 

Remarks: (frustration level, clarity 
of directions) 



B. Attitude Toward- Materials 

1. Participant comments about difficulty level of material: 



2. Participant conments about utility of material: 
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II. pr£ and post tests with answers 
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PRETEST: EVALUATING INSTRUCTIONAL PROGRAMS 
Prototype Testing 



Direct tons" ; Read the description of. the materlnl which follows 
then answer the questions below In as few sentences as possible. 



Materials Description 

An experienced master teacher Is developing a series of materials she 
hopes will teach elem'entary school children to read music more easily 
than existing state adopted t6xts. The components she has planned are 
as follows: 

- a fllmstrlp and accompanying tape which depicts musical notes 
as talking cartoon characters, 

- a pupil booklet which includes simple sheet music, a short 
text, ^and plastic overlays displaying various types of 
musical notes and symbols, \^ 

- a teacher's guide keyed to both the fllmstrlp and bookl^^j,. 

- all materials are presently In rough form, 

ASSUME IT IS YOUR JOB TO "PROTOTYPE" TEST THESE IJATERULS, THE INFORMA- 
TION GATHERED FROM THE TEST WILL PROVIDE THE BASIS FOR REVISION OF THE 
MATERIALS, ANSWER THE FOLLOWING QUESTIONS AS IF YOU WERE PLANNING SUCH 
A TEST, s,^ 

1, What information about the materials do you wish to acquire during 
the test? 



2; What kinds of subjects will you use? 
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NAME 



Prototype testing (cont.) 

3^ What measures or data sources will you use to obtain the infornatinn 
vou desire? 



T 



Wlnat preparfttioi 
runs snoothly? 


11^ will you make in advance to make sure that your test 














1 * 


■ . 1 

» 

/ 

* 


\ 

t 

• 
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PRETEST CONFIRMATION SHEET 
Data Sources 



The responses sup,j»csto(1 herein are deslr.npd to be Tcpresrntatlvo rntber 
than exhaustive examples of adequate ways In vhlch each question night 
have been ausuered. 

Question 1 

All responses should focus on acquiring Information from or about 
students rather than teachers. 

SAMPLE RESPONSES: ' 

a) collect criterion data 

Did students attain the objectlve(s) ? 

collect practice (wlth£n-program) data 

How well did students perform on en route objectives? 
How well did students perform on practice exercises? 

collect affective data 

Did students feel they understood the Instruction? 
Did students like the program? 
Were materials ^too difficult or simple? 

collect observer data 

Did students use the materials In the prescribed manner? 
Was there evidence of frustration or other difficulties 
In using the materials? 

» 

Question 2 

A small number of students (no more than 10-12) selected from all 
grade levels In which the materials are designed for use, 

Question 3 

• Pretest data Is not necessary If baseline data has been previously 
collected. Responses should focus on students. 

SAMPLE RESPONSES: 

a) practice exercises within the program completed by student 

b) criterion test (posttest) 
, c) attitude survey 

d) Interview forms 
«) observer notes 



b) 



33 



Confirmation Sheet (cont.) 
\ 

ijiiestt on % 

Suggested preparations for a smoothly administered test Include 
the following: 

■9 

a) secure permission to administer test 

b) secure a location with a proper testing environment 

c) train observer(s)/lntervlewer(s) 

d) provide extr^ copies of test materials 

e) double check materials for errors 

f) ensure names are Included on products or written materials, 
design coding method if names are omitted 

g) review directions with all persons Involved in administering 
the test ' ■ 



I 
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PRETEST 

Rovisiiif* Instructional Programs 

Directions ; You are provided with the following development Information; 

1. description of targe|i population 

2. description of materials 

3. terminal and sub-objectives 

4. terminal objective item form 

5. raw prototype test data from four sources 

YOUR TASK: 

1, Plot criterion test and subobjective data using frequency 
polygons, 

2. Identify data patterns for terminal and subobjectives. 

-3. Suggest revision decisions consistent with research-based 
principles and product specifications. 

Describe how you would implement your recommendation(s) for 
revision. 

5. Be sure to write vour name on the answer sheets. 
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PRETEST 



"Discover Luf, Vnur Philosophy" 
' PRODUCT SPECIFICATIONS 

• Tarp^ct Population: "Discovering* Your Philosophy" is ^iii "applied i)hllasoph 
course to offered to senior high school students. The course Is 
designed fot students who wish to clarify their own values on con^ 
troverslal issues and be exposed to various philosophical viewpoints. 

Description of Materials; 

1. Hidden Feelings — a short programmed book with guidelines and 25 
exercises to train students in identifying value terms and phrases. 

2. "What Do You Think?" — a pamphlet containing a series of Issues 
about which students must write a one paragraph value statement. 

3. Four Great Phllosophers -^four short booklets providing thumbnail 
analyses of the major philosophical Ideas of Plato, Rousseau, 
Aristotle and Marx* Each booklet contains ten exercises in which 
students must discriminate one philosopher's assumptions from 
those of other philosophers* 

4. An attitude questionnaire. 
Terminal Objectives: 

Given a political, social or economic question about which a policy 
decision must be made, the student will 

1) identify the value issues of the policy question 

2) state his/her own values 

3) indicate an appropriate historical or philosophical referent 
for each of the values mentioned. 

Subobjectlves: 

1. Student will identify value terms included in position statements^ 

2. Student will write a short position paper recommending a policy 
decision on a given issue, using connotative words and phrases. 

3. Student will identify value terms included in position statements. 
Terminal Objective Form; 

Response Description : The student will write an essay or tape record 
a response which specified: 

1) three valine issues of the policy question 

2) a summary of personal values with regard to the policy. 

3) historical or philosophical referents for each of his/her value 
statements. 
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Cont«>t Limits ; The learners will be 8^^^ » "j^^^ P^^^X^Lml 

«) the American economy, b) personal moral bdiavlor. ^> 

relations, d) political actions, e) penal reform, f) 

euthanasia, g) treatment of dlsaldent groupa. and h) Wf-tment of 

minority groups. Student vlll select one policy from the above 

upon ^Ich to base his/her response, 

criteria ; Identification of value Issues meana the J*;^^*'^ 
list any couraea of action and/or queatlona that include J«j8in«ntal 
terms, k susnary of personal valuea vlU Include an ••»«tlon of «hat 
la rl^t. desirable. Important, necesiary or the converse. HUtorl- 
cal and philosophical documentation for these "^^i^. ^1^^ 

drawn frSm a) philosopher, dlscuased In class, b) P'«i««»^ '^f '^'^ 
for certain cauaea drawn from the press or national T.V.. or c) any 
published interpretation of historical events. Documentation must be 
given for at least five assertions. 

Outline of Prototype Test Procedures; 

Four high school students participated In prototype P^^'^l""- 
Studenta read and completed practice exercises for each of the fiESfil 
g£aSonhera booklet^ then discussed their "-^^^i"?- /j;^ "-J^^^ 
the e«e!:clse s In class in order to obtain feedback for th^lf^^- 
•^at Do Tott Think" and Hidden Peelings were completed ••^«»«*»»"^ 
aaalKnmenta. while the rest of claaa time was devoted to discussion 
" p™J value, aa well a. those of the Ph^l<>-f SJaSs- 
the materlala. Students were encouraged to Introduce Idea, of P^^os 
JpJe« and hUtorlans not covered In claaa. Finally. 
the criterion task. Only one student chose to record, rather than write, 
«:pS^e: to the eU. UPon the completion of 

completed an attitude questionnaire and then held a "round table dla 
^Slon about the -terlaU. The Instructor kept note, of pertinent 
rcmarka made during thla dialogue. 
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PROTOTYPE TEST DATA 



Criterion Test Data 



Teriairral Objective #1 
^^Xid^^tify 3 value Issues) 



Student 

Clancy 
Siegfried 
Mathilda 
Yvonne 



Score 

1 
3 
2 
1 



Terminal Objective #2 
(State own values) 



Student 

Clancy 
Siegfried 
Mathilda 
Yvonne 



Adequate 
Response 



X 
X 
X 
X 



Inadequate 
Response 



Terminal Objective #3 
(Indicate value referents 
for at least 5 assertions) 



Student 

Clancy 
Siegfried 
Mathilda 
Yvonne 



Score 
1 

2 . 
4 

1 



Subobjective Data 



Subobjective i^l 
(identify value terms 
—25 pts. possible) 



Subobjective 
(vnrite position paper) 

adeqt. inadeqt. 



Subjective #3 
(identify sources 
"40 pts. possible) 



Student 


Score 


Student 


respon. respon. 


student 


Score 


Clancy 


20 


Clancy 


X 


Clancy 


30 


Siegfried 


17 


Siegfried 


X 


Siegfried 


27 


Mathilda 


12 


Mathilda 


X 


Mathilda 


18 


Yvonne 


15 


Yvonne 


X 


Yvonne 


22 



Results of Questionnaire 



Agree 



Disagree 



I enjoyed the sequence of values 


4 




I found Four Great Philosophers easy to use 


. 3 




I thought 'Vhtt Do Tou Think?" was helpful In 

clarifying my own values. 


4 




Hlrlden Feelings helped me learn about words 
that Imply values. 


4 -V 





38 



The following coinrionts are representative of cuinnents made during the 
round t^ble discussion^ 

' * 

"I really enloyod " this prop,rari." 
"I'd like to do It again." 
• "Even thoufjh I didn't well, It was fun," 
"Some of the exercises were too difficult." 
"I never seemed to quite catch on," 
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PRE- TEST ANSWER SHEET 



pirectlons ! Plot tertnihal objective data, using frequency polygons. In 

tHtf spaces provided below. Describe data patterns on the line below each 

graph. Plot aubobiective data in the same manner but on the next page, 

BE SURE TO LABEL EVERY GRAPH. After you have finished, go on to the following 

pages to suggest revision and implementation procedures. 



CRITERION TEST RESULTS 
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Data Pattern: 



Data Pattern: 



Otta P«tt«ra:_ 



40 



SUBOBJECTIVE RESULTS 
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. — < 


♦ 

-i 


4 


• t 

. i 

1 


-T 

4 

t 


•1 
— ♦ 


X 


■ -i 

■i 

- — « 


•1 ■ 1 
-1- ^ 
i-t 


1 

:] 


T 

t 

4 


T 

- 1 

•1 


• < 

1 

-i 
— t 




1 


^ 1 1 ♦ 

f-! 1 1 

-t— t t T 














1 i 






1=1 






































































— ♦ 


-i 










; T 
































- 



































1 


1 




— 4^— . 










1 


1 












■-h 




1 








1 




\ 














— 1 












1 ^ 


L J , 






-J 








— i — 1 






I . ; 






. I 




































1 












— , 




p — ' 




— 1 


I 

i 










— * .. A.— 1 












1 




' 


1 — 


■4- 










-4- 




















1 _ 












i 




-- 












1 






— 1 


1 










4 — 






4- 








1 . 










■ , i j ^ 


















— 1 — 1 


[- 


1 








1 — ' 




I— 






-^ 
























r 


T 




1 








k— — 4— 


















— ^ -t 


T 










































1 — 


1 — 


t 
























t 



















































Data Pattern: 
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Data Pattern:. 
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Pie test Answer Sheot (cont.) 



2. Which research-based revision principles would you use to modify this 
instructional product? Mark an X by all revision^ that apply. 

Terminal Obiective tfl> Suboblectlve #1 

add practice 

review task analysis 

verify unprompted practice 

. pool and redistribute practice and posttest items 

add feedback 

provide task description (assure learner understands task) 

. provide motivational stimuli 
^^^^^ carefully delete Irrelevancies 
J.eave everthlng alone 
^change format 



Terminal Obiective it2. Subobjective #2 

add practice 
review task analysis 
verify unprompted practice 

pool and redistribute practice and posttest items 
add feedback 

provide task description (assure learner understands task) 
provide motivational stimuli 
carefully delete Irrelevfincies 
leave everything alone 
^change format 



Terminal Objective ^^3, Suboblectlve 

add practice 
review task analysis 
verify unprompted practice 

pool and redistribute practice and posttest Items 
add feedback 

provide task description (assure learner understands task) 
provide motivational stimuli 
carefully delete irrelevancies 
leave everything alone 
^change format 



Please go on to the next page. 
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Pretest Answer Sheet (cont^) 

3 How would you mocHfy specific Instructional activities listed In the 
* specifications to apply the prlnclple(s) you have selected above? 
' (e.g., "rewrite %he directions "in the Teacher s Guide ) 



r 
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PRETEST CONFIRMATION SHEET 



H<;\/lsinv; Instructional Pro*»ra!ns 



Criterion »TesC and Subobjectlve Data 



.. Terminal Objective f/l 



Terminal objective #2 



'i 



1^ items correct 




adequate 



Inadequate 



Terminal Objective #3 



BEST con miAt:: 




0 correct Items 
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Preteat Confirmation Sheet (cont.) 



■srnnffinuBfE 



Subobiectlve #1 



Subohjectlvo if2 



4 



TT 



1 



mm 



# points attained 





"1 1 






























i . 
































ff 







points attained 



Subobjectlve #3 




adequate Inadequate 
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Pretest Conflriiiat Ion Sheet (cont.) , . 

■ 

2« The research"* bas)ed revision principles which should be indicated 
are given below: ? 

Terminal Objective #1, Subobjective iPl 



X add practice ' 
X review task analysis 
^^^^^ verify unprompted practice . 

^ pool and redistribute practice and posttest items 

X add feedback 

X provide task'^'^^scrlption (assure learner understands task) 

provide motivational stimuli 
. carefully delete irrelevancies 
leave everything alone 

^change format 



Terminal Oblective #2, Subobjective #2 

add praictice ^ . \ 

review task analysis , ^ 

. verify unprompted practice 
_ pool and redistribute practice and posttest Items 

add feedback 

provide task description (assure learner umferstands task) 

provide motivational stimuli 

JC carefully delete irrelevancies 

X leave everything: alone 

^^^^ ^change format 



Terminal Objective <^3. Subobjective #3 

X add practice - 

X review task analysis 

verify unprompted practice 

pool and redistribute practice and posttest items 
X add feedback ^ 

X provide task description (assure learner understands task) 

provide motivational stimuli 

carefully delete irrelevancies 
^^^^^ leave everything alone 
' ^change format 
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Pretest Confirmation Sheet (cont.) 

3 SuRgeptions for implementing the revision principles In question 2 

might be as follows: (responses are Intended to be merely suggestive) 

Terminal Objective #1, Subobjectlve #1 

Add^ feedback to Hidden Feelings . 

For example, the booklet should be returned as soon ao possible 
f ^following Its completion by the student. ^ 

Add more practice and/or exercises to the Hidden Feelings booklet , 
to ensure that the students have adequate opportunity to become 
familiar with the task. 

Make sure students understand criterion task for the terminal 
objective of what Is expected. This might be done orally by the 
Instructor In class, or amended to the booklet, and position state- 
ment papers. 

Review task analysis of requisite skills. 

Doss TH ^i^«»n ffAglinPfi omit any Important training In helping student 
Identify value terns and phrases? Is adequate practice given for 
the terminal objective? 

■ ' j ' • ' 

i ■ ■ 

Terminal Objjectlve #2, Subobjec. lve #2 

All materials should be lefr. as thev are . Although It would 
be tedhnlcally correct to suggest deleting irrelevancles ftom 
••What'Do You Think?" the subject matter would seem to make 
this task an exceedingly difficult one. 

Terminal Objective #3, Subobjectlve ^3 / . . 

Add more opportunities, for student practice in identifying 
the viewpoint of a given philosopher, e.g.\ more structured 
discussion practice, short quizzes, additlcjnal/ written, 
exercises. 

Make sure students receive feedback for philosophical referents. 
For example, discussion of responses to exercises contained In 
Four Great Philosophers may not provl4e sufficient feedback. 
One might add written" Confirmation' fo accompany the exercises 
then test again to see If the problem has been resolved. 
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Pretest Confirmation Sheet (cont.) 



Make sure students understand the task . Perhaps the 'directions 
In Four Great Philosophers are not well x^itten. In addition to 
clarifying extant directions, a further description of what the 
students are to do might be given orally by the Instructor during 
class time, 

Tleview the task analysis for Four Great Philosophers . It is 
possible that the skills required in Objective #3 are built upon 
prerequisite training not provided in the instruction. Admittedly, 
given the subject matter of this material, task analysis may be a 
difficult procedure. 
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POSTTEST: EVALUATING INSTRUCT lOlJAL PROGRAMS 



Prototype Testing 



Dtrectitons ; Read the description of materials given below. Use tl 
worksheet to write an outline of the major elements of a prototype 
test plan you would design^ for the materials. Be' specific In your 
use of data sm.rces. measures, sub j ects and provision for admlnlst 
tlve Issttes-. Be sure^ to write voqr name-on th^ worksheety- 



Materlals Description . . 

A uaior difficulty experienced by students at the graduate level centers 
upon writing a dissertation proposal. Ttiree professors In the Education 
Department are attempting to reduce this problem by developing materials 
which will help graduate students develop skills relevant to this sanc- 
tlmes traumatic task. .Reugh models of the materials are now ready for 
testing. The components of the instructional kit are as follows: 

"The Harrassed Students' Guide to Literature Review" a self- 
instructional booklet which provides practice in the skills 
necessary for a thorough literature review 

a film entitled "Scratching Your Intellectual Itch" which suggests 
hunian resources for review anci- improvement of a proposal 

simulations (five) called "The Pitfalls of Procedures" which 
require participants to criticize quantitative data gathering 
techniques and suggest alternative strategies for given research 
situations 



The objectives of the products can be\lnferred from the component titles. 



'I 



On the next page, outline your proposed prototype test plans for 



these materials^ 
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POSTTBST: BVALUIITIHG niSTRUCTZONAL PROGRAMS 

Work«h«et 
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POSTTEST: EVALUMING INSTRUCTIONAL PROGRAMS 



Revision . 



Directions: You are provided with the following development information: 
1. description of target population 



3. terminal and subobjectlvas 

4. terminal objective item form 

5. raw prototype test data from four sources 

YOUR TASK: 

1. Plot criterion test and subobjectlve data using frequency 
polygons. 

2. Identify data patterns for terminal and subobjectives. 

3. Suggest revision decisions consistent with research-based 
principles and product speclficav-tons. 

4. Describe how you would Implement your recotmncndatlon(s) 



2, description of materials 



for revision. 



BE SURE TO WRITE YOUR NAME ON THE ANSWER SHEETS PROVIDED. 




^^STTEST: REVISING INSTRUCTIONAL PROGRAMS 



••Ecology In Action*' 



PRODUCT SPECIFICATIONS 



Target Population: American homemaker 
Description of Materials: 

1. Television broadcasts (in rough form) covering the following 
topics: 

a. Crises Past: An Overview . 

b. Principles of Ecological Living 

c. Crises Present 

What Can W£ 'Ool (group action strategies) 

e. What Can I Do? (individual action strategies) 

f. Read the Writing on the Wall: Beating the Pinch 
g« What Have You Learned? (criterion examination) 

2. Participants* self^instructional workbook, containing exercises 
keyed to each telecast « 

3. Attitude survey forms for each participant. 

Terminal Objectives: 

Participant apply principles of ecological living to family 

management by: 

1. identifying in writing three inminent crisis situations 

2. writing at least two original individual actions that would 
ease a specified crisis in his/her home. 

Subobjectives: 

1. identify five principles of ecological living 

2. orally suggest two group strategies for coping with a 
given ecological crisis. 

Termina^l Objective Item Form: 

Response Description ; Given the conditions likely to precede 
three types of ecological crises, the participant will 1) identify 
the implied crisis and 2) write at least two original strategies 
for coping with each crisis at an individual level. 
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Cotuent Limits ; The participant will be shown portfolios of 
hypothetical news clippings, public service ads, and political 
cartoons pertaining to three crises: fuel shortage, protein 
supply shortage, pollution. Students will select one of the 
three portfolios; 

Criteria : An adequate response contains these elenents: 

1) Identification of probable crisis by listing effects of 
current situation. 

2) supporting evidence of effects taken from documents. 

3) two strategies for coping with the crisis that have not 
been mentioned in class. A "coping" strategy is one that 

a) utilizes solutions directly related to the cause of 

the crisis " . . e u 

b) ^ takes individual responsibility for alleviation of the 

crisis rather than depending on outside agencies for 
help. . ^ 

i. 

Outline of Prototype Test Procedures 

.The broadcasts were sho^^ to eight male and 'female volunteers in a 
homemaker's "ecology workshop" offered In an Adult Education Course. 
Participants viewed all broadcasts and completed workbook exercises 
for each During the seventh broadcase. the participants completed 
an exam then responded in writing to a series of ftltude survey 
questions about the usefulness and enjoyabllity of the series. 
Following each of the sessions, two different participants x^ere ^ 
interviewed by the Instructor and gave their opinions of the evening 
Instruction. At the end of the course, all materials were turned In 
to the Instructor and returned to the developers of the series. 



53 



PROTOTYPE TEST DATA 



Criterion Test Results 



Terminal Objective #1 
Identify Crisis Situations 



Terminal Objective #2 
Suggest Individual Strategies 



Participant 

Raquel 

Christopher 

Richard 

Margaret 

Elizabeth 

Vlckl 

Dennis 

Susan 



(3 possible) 

# correct 
Identifications 

3 
3 
3 
3 
2 
2 
3 
3 



Participant 

Raquel 

Christopher 

Rlchar^ 

Margaret 

Elizabeth 

Vlckl 

Dennis 

Susan 



(2 possible) 

^ adequate 
strategies suggested 



2 
2 
2 
1 
2 
2 
2 
1 



Suboblectlve Data 

Subobjectlve #1 
Identify 5 Principles 

(5 possible) 

# principles 
Participant correctly Identified 



Raquel 


2 


Christopher 


3 


Richard 


4 


Margaret 


4 


Elizabeth 


2 


Vlckl 


5 


Dennis 


3 


Susan 


4 



Subobjectlve #2 
Suggest Group Strategies 

(2 possible) 

^ adequate 
strategies suggested 

0 
1 
2 
1 
0 
1 
2 
1 



Participant 

Raquel 
Christopher 
Richard 
Margaret 
Elizabeth 
Vlckl 
Dennis 
Susan 



Results of Attitude Survey 



I learned specific ways to copy with probable 
crisis situations. 

The programs held my Interest all hour. 

I would recommend the program to other home- 
makers. 

I would like to see other programs developed 
by this company. 



Agree 

7 
3 

2 
0 



Disagree 

I 
5 



8 
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POSTTEST ANSWER SHEET 



\ 



Dlrfc-ttons : Plot terminal oblecttvc data, using frequency polygons, in the 
spaces provided below. Describe data patterns on the line beside each grdph. 
Plot subobiective data in the same manner but on the back side of this pace. 
3E SURE TO LABEL EVERY GRAPH. After you have finished, go oa to the nex»-. 
page to suggest revision and implementation procedures. 



CRITERIQ1L_TESX_RESULTS 
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Data Pattern; 



Data Pattern; 
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SUBOBJECTIVE RESULTS 

Be sure to label graphs. 
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Data Pattern: 



Data Pattern: 



NAME 



PosLtest Answer Sheet (cont.) 

2 Which research-based revision principles would you use to modify this 
' ' instructional product? Mark an X by all revisions that apply. 

Terminal Objective H, Subobiective #1 

add practice 

review task analysis 

verify unprompted practice 

^ pool and redistribute practice and posttest Items 

add feedback . 
provide task description (assure learner understands task) 

provide motivational stimuli 

■ carefully delete Irrelevancles 
leave everything alone 

* change format 

Terminal Objective #2, Subobiec tive ^ 

add practice 

review task analysis 

verify unprompted practice 

pool and redistribute practice and posttest items 

add feedback ^ . ^ i n 
[ provide task description (assure learner understands task) 

provide motivational stimuli 

carefully delete irrelevancles 

leave everything alone 

•• * change format 

n 

3. HOW would you modify specific instructional activities listed in the sped 
flcatlons to apply the prlnciple(s) you have selected above? (e.g., re 
write the directions in the Teacher's Guide") ^ 



(Use the other side of this page if necessary) 



erJc 



«aot • retearch-btied strategy for revision. 
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P0STT2ST COMFIBMATION SHEET 
Revising Prograas 



1. 

Terminal Objective #1 
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# correct lt« 
(wide range data pattern) 
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ofjltest Confirmation (cont.) 



Althouj^h {>erfornauct\ on^Suuohject Ives 1 and ? vvas less than outstanding, 
Ijotli Terminal Chjeciives were attained by tuost participants. Results 
of the attitude survey indicate an obvious lack of interest in the pro- 
gram. Therefore, your response should have been as follows: 

Terminal Objective #1, Subobjectlve #1 

X provide motivational stimuli 

Teruinal Objective #2, Subobjectlve #2 
X. provlfJo motivational stimuli 

. / 

/ 

Perhaps the presentation of the Instruction is overly depressing due to 
the serious nature of the subject matter. One m.lght attempt to add humor 
to the broadcasts wherever possible. Another way to approach the proMem 
of motivation might be from the standpoint of group interaction. The use 
of the materials Included no discussion among participants and nay have. 
cause<' them to feel Isolated or bored, A "round table" follow-up to each 
audiovisual presentation might help alleviate disinterest In the program. 
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/ PARTICIPANT QUESTlbNNAIRE 



>v lo order to ln{vrov< 
your xyj^uHd responses t( 



ove the natGrl^ls you have usetl, we would appreciate 
o the following questions. Please feel free to 
write additional comments on the hack of this page. 

DO NOT WRITE YOUR NAME ON THIS PAPER 

A. Circle the appropriate number. 

1. Concepts were clearly presented In the text, 4 3 2 1 

2. New Ideas were Illustrated by an adequate 4 3 2 1 
number of examples. 

3. Directions for practice exercises were "^4 3 2 1 
clearly written and easy to follow, 

4. Confirmation sheets were accurate and com- 4 3 2 1 
plete. 

5. Posttests were fair and of an appropriate 4 3 2 1 
level of difficulty. 

6. The Data Sources (yellow) Section held my 4 3 2 1 
attention. 

7. The Data ^Summary (buff) Section held my 4 3 2 1 
attention. 

8. The Revising Programs (blue) Section held 4 3 2 1 
my attention, 

9. I feel the Information presented in this pro- 4 3 2 1 
gram will be useful to me at a later time. 

B. Briefly 'answer the following questions: 

10. In What- ways do you think these materials can be improved? 



11. What do you consider to be the strong points of this program? 



12, Would you recommend these materials to another educator? Why or 
why not? 
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III. GLOSSARY 
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ERIC 



BiST COPY AVAILABLE 



GLOSSARY 



^>i-pu).!<->l (list rthution ; a tally or graph that has two modes, that is, two 
scores which occur west frequently In a distribution^ For cxanp^^-» 



frequency 







* ' ■ * ^ f 

1 




I f ' ' 


















1 






1— ^ — y 


















































































































































































h 


















\ 
























































































L 






















>- 




I 




-i 










>- 






< 

























til 



0 correct items 



compon-ent : a part of an instructional product such as a text, game, film, 
teacher's guide, answer book, etc« 

« 

criterion test : a test given at the end of an instructional sequence to 
measure student performance on the objectives of that pequence« Items 
on the test are keyed to specified objectives stated in terms of stu- 
dent behavior^ 

data ; information about the effectiveness of a program. Data may be quan- 
titative, as in the form of test scores, or qualitative, as In student 
comments, written responses to questionnaireis or oblserver reports^ 

data sources : the types of information acquired about a product « 

data conflicts ; information about the effects of a program that indicate 
an apparent discrepancy in the instruiftion. For example, postteat 
scores may indicate that student achievement is high, while attitude 
survey shows the students detested the program^ 

entry skills: those behaviors idiich a student must have mastered prior to 
the onset of an instructional sequence^ Entry skills for a program 
that teaches students to multiply one digit numerals might be that stu- 
dents can count to 100 and add any numerals whose sum la less Chan 100« 

feedback : .information, given to a student about the adequacy of his/her re* 
sponses. Feedback can be as simple as '*that*s right** or as complex as 
a detailed theoretical explanation of an algebraic postulate; 

formative evaluation : gathering information about the adequacy of a program 
for the purpose of revising and Improving the program. 
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BEST COPY AVAIUBLE 

/ 



Glossary (continued) 



frequency polygon ; a simple line graph which Indicates the number (fre- 
quency) of correct student responses to a quantity of items. For 
example: • 



number of 
students 
(frequency) 
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2 students scored 8 
2 students' scored 9 
2 student^ scored 10 



# correct responses 

histogram ; a simple bar graph often used to display thji same information 
as a frequency polygon. For example: 



number of 
students 
(frequency) 



10 
h 




b A ^ G 

# correct responses 



2 students scored 8 
2 studenils scored 9 
8 students scored 10 



\ 
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ERIC 



BEST COPY AVAILABLE 



Glossary (continued) 



homogeneous data : Information about the effects of a program which Is all 
of a similar nature. For example: 



Graph A 



frequency 




Graph B 



frequency 
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# Items correct # Items correct 

Graph A exemplifies homogeneously negative data. 
Graph B exemplifies homogeneously positive data. 

instructional development ! a process of creating educational materials that 
is based on a cycle of tryout and revision. The framework of this pro- 
cess might be sunonarized as follows: 1) write a description of what 
the materials will teach and the methods to be used in schieving the 
objectives stated (specifications)* 2) create a test to measure the ob- 
jectives, 3) try out tlv test to ensure students cannot accomplish the 
objectives without instruction, 4) revise the criterion test if neces- 
sary, 5) generate a rough draft of the materials, 6) try out the rough 
materials to determine if they are effective in teaching the objectives, 
7) revise the materials as needed, 8) try out the revised materials with 
a larger group of subjects to ensure they are effective with the type of 
student for which they are designed, 9) revise and retest if necessary, 
10) disseminate. 

item form : a model that shows how items on a criterion test are constructed 
so that another developer can create equivalent test items. Item forms 
Include a description of an acceptable student response, the circumstances 
under which the student will be tested, the limits of the content of the 
item, and ways in which an instructor would differentiate correct from 
incorrect responses (criteria). 



lean strate^ for data collection ; a process by which a product developer 

focuses on acquiring only that information which is absolutely necessary 
to the improvement of a set of materials, e.g., using small numbers of 
students during orototype testing, collecting only that data which will 
be relevant to revision, focusing on specific modification decisions to 
be made about the materials. 
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Glossary (continued) 

mean ; the average; the sum of scores divided by the number of scores. 

measure: Instrument used to obtain Information about the effectiveness 
of a product. For example: tests, questionnaires, observation 
forms, Interview blanks. 

objective : a ststement of what the learner Is to do as a result of an 
Instructional sequence* 

*• 

en route objective ; a statement of **at a learner must^ able to 
do as a prerequisite to accomplishing a terminal objective. 

' t 

subobjectlve : same as an en route objective. 

terminal objective : a statement of what a learner must be able to 
do as the final result of an Instructional sequence 

outcome : results of Instruction, planned or unplanned. 

practice ; student trials of the skills required by the objectives of a 
progrsm. Prsctlce may take the form of written exercises, role- 
playing, oral recitation, etc. All practice should be either an 
analagous or Identical to the task which will ultimately be performed 
by the student at the conclusion of Instruction, 

prerequisite skills : same as entry skills or entry behaviors. 

product ; any set of Instructional materials with specified objectives, 
provision for learner practice and some form of criterion test. 

product developer ; a person who directs work on or creates Instructional 
products. 

prototype test ; a tryout of rough draft products or parts of products 
(components) using a small number of subjects. The purpose of a 
prototype test Is to determine whether the learning sequence that 
has been designed Is functioning, 

reactive pretest : a test given prior to Instruction which affects later 
student performance. For example, a pretest which assesses student 
attitudes toward the use of drugs may sensitize participants to the 
Information the developers wish them to acquire. Student responses 
In latter parts of the Instruction may be affected by the students' 
attempts to respond In the way i^lch Is perceived to be "expected." 

esearch-based Instructional principles ; generalisations about how people 
learn which are Inferred from empirical (experimental) research. 

repUcable : able to be reproduced In another setting. Instruction prod- 
ucts should be tepllcable, In that their effectiveness should not be 
tied to a single teacher or classroom. 
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Glossary (continued) 

specifications ; a description of the plans for an Instructional product 
(soe "Instructional development^ " 

subjects : the group of learners with which an Instructional product is 
tested. Subjects should be comparable in age and experience to the 
group for whom the materials are ultimately Intended (target population). 

su^Tiinative eva^luatlon : an assessment of the adequacy of an Instructional se- 
quence, x^lth no Intention of revision. 

rar5-ct population : the group of learners for which a certain set of materl- 
als are designed. A target population may he any group from "house- 
wives'^ to "third year medical students specializing in endemic diseases." 

task analysis : a process by which an objective Is "broken daw»\" into all the 
siibtasks that might be necessary to master the final behavior. The sub- 
tasks Ideally are verified by testing the sequence to ensure that no 
essential *'steps" are missing. 

wlde-range data : Information which suggests no single trend In results; 
the opposite of homogeneous data. For example: 
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correct responses 



No positive or negative 
trend is Indicated by the 
scores**they are distribu- 
ted across the graph. 



with in- program responses: answers provided by students during an Instruc- 
tional sequence as they practice for a terminal objective. Within- 
program responses may assume various forms, depending upon the nature 
of the practice provided, e.g., discussion questions, written exer- 
cises, etc. 
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